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Conceptual  Flow  of  the  ATAM 
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Data  Set 

18  ATAMs 

•  12  DoD 

•  2  other  government  agency 

•  4  commercial 

The  ATAMs  were  performed  between  2000-2005. 
Domains  range  from  embedded  to  information  systems 
137  Business  Goals 
99  Risk  themes 
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Affinity  Diagram 

Bottom  up  process  to  discover  groups  in  raw  data 
Developed  by  an  anthropologist 
Relies  on  intuition 

Two  data  items  are  in  the  same  group  if  the  grouping  team  feels 
they  have  something  in  common 

A  data  item  can  be  placed  into  multiple  groups 

Groups  are  then  categorized  based  on  judgment,  literature. 
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Risk  Theme  Categories 


Risk  themes 
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Interesting  risk  themes 

Exhibited  by  over  50%  of  ATAMs 

•  Performance 

•  Requirements  uncertainty 

•  Lack  of  addressing  important  considerations  (samples 

•  Organizational  awareness  on  next  slides) 

Documentation 

•  Occurred  in  exactly  last  5  ATAMs 

•  May  be  due  to 

-  Increased  sensitivity  on  part  of  evaluation  team 

-  Better  documentation  of  system 
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Sample  risk  themes  -  addressing 
important  considerations 

There  are  many  risks  arising  from  decisions  not  yet  made.  The  volume  of  decisions 
not  yet  made  suggests  that  the  project  schedule  is  at  risk. 

There  is  a  lack  of  support  for  data  management:  There  is  no  uniform  specification 
for  managing  meta-data  and  its  persistence.  There  is  no  strategy  for  ensuring  that 
data  sets  are  accessible  outside  of  an  implementation  of  a  sub  domain.  This  means 
that  while  data  is,  in  theory,  exchanged  by  all  sub  domains,  they  may  not  be  sharing 
the  same  assumptions  about  the  data.  And  it  may  not  be  easy  for  one  sub  domain 
to  gain  access  to  data  sets  from  another  domain. 

There  is  a  trend  to  move  toward  an  integration  role  for  the  development 
organization.  This  increases  exposure  to  liability  risks  in  customer  and  3rd  party 
software  integrated  with  development  organization  software.  The  market  is  forcing 
the  development  organization  to  be  an  integrator,  but  there  is  no  clear  business 
goal  that  states  this. 
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Sample  risk  themes  - 
organizational  awareness 

There  are  risks  arising  from  a  lack  of  an  adequate  training  program  especially  for 
the  pool  of  developers  that  will  be  implementing  the  system  under  review 

The  new  architecture  may  not  be  institutionalized  for  two  primary  reasons: 

1.  Not  everyone  is  sensitive  to  the  benefits  that  the  architecture  can  offer. 

2.  The  guidelines  and  rules  for  developers  regarding  when  to  use  which 
architectural  mechanisms  are  not  complete  yet. " 

The  new  component-based  product-line  approach  provides  extensive  potential 
which  cannot  be  exercised  without  training,  application  development  guidance, 
and  tool  support. 

There  is  a  lack  of  attention  to  support  and  training  issues  in  the  architecture  of  the 
system  under  review. 


There  is  a  test  requirement  to  interoperate  with  other  systems  but  neither  test  plan 
nor  test  capabilities  have  been  detailed  beyond  those  internal  to  the  system 

under  review 
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A  Different  Categorization  of  Risk 
Themes 

Risks  of  commission  -  those  risk  themes  that  refer  to  a 
decision  in  the  architecture  that  is  problematic 

Risks  of  omission  -  those  risk  themes  that  refer  to  the  lack 
of  a  decision  or  investigation 

Other  -  those  risk  themes  that  are  neither  commission  or 
omission 

Commission:  25  of  99 

Omission:  57  of  99  (inter-rater  reliability  test  is 

Other:  18  of  99  .82) 
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Risk  Themes  Categorized  by  Omission 
and  Commission 
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Possible  factors  to  predict  risk  themes 


Came  to  the  SEI  -  not  a  random  sample  of  systems  by  any  means 


Business  goals  -  e.g.  do  systems  with  performance  as  a  business  goal 
have  performance  risks? 


Domain  of  system  -  e.g.  do  embedded  systems  display  different  set  of 
risk  themes  than  information  systems? 


Dominant  architectural  style  -  e.g.  do  client  server  systems  display  a 
different  set  of  risk  themes  than  cyclic  executives? 


Evaluation  team  -  are  risks  result  of  examiners? 


Development  team  -  maturity  of  team,  size  of  system,  skill  set  of  team? 
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We  have  explored  two  possible  causes 
for  risk  theme  patterns: 

Business  goals 
Domain 


In  each  case,  we  are  looking  for  patterns  in  risk  themes 
that  share  either  business  goals  or  are  in  the  same 
domain. 
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Business  goal  categories 
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Business  goal  distribution 
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•business  goals 
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Do  systems  with  performance  as 
business  goals  exhibit  higher 
probability  of  performance  risk? 


•Symmetric  Measures 


•Value 

•Asymp. 
•Std.  Error3 

•Approx.  Tb 

•Approx.  Sig. 

•Interval  by  Interval 

•Pearson's  R 

•.194 

*.233 

•.792 

*.440c 

•Ordinal  by  Ordinal 

•N  of  Valid  Cases 

•Spearman  Correlation 

•.194 

•18 

*.233 

•.792 

*.440c 

•a. *Not  assuming  the  null  hypothesis. 

•b. 'Using  the  asymptotic  standard  error  assuming  the  null  hypothesis, 
•c. ‘Based  on  normal  approximation. 


NO! 
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How  about  domains? 
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Identified  the  following  domains  in  the 
1 8  ATAMs 


Domain  Number  of 

ATAMs 

Avionics  3 

C4ISR  1 

Command  and  control  4 

Command  and  Intelligence  1 

Distributed  infrastructure  1 

Embedded  information  systems  2 

Embedded  control  systems  2 

Information  Systems  1 

Information,  Surveillance,  Reconnaissance  1 

Mission  computing  1 

Modeling  and  simulation  1 
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Do  systems  from  the  same  domain 
exhibit  a  pattern  of  risk  themes? 

For  domains  with  more  than  1  ATAM,  we  calculated  a  measure  of 
similarity  of  risk  themes.  We  are  still  thinking  about  what  constitutes 
a  good  measure  of  similarity  (.7  means  significant  similarity  for  the 
measure  we  are  using.) 
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Domain 

N 

Measure  of 
similarity 

Avionics 

3 

.245 

Command 
and  control 

4 

.131 

Embedded 

nformation 

system 

2 

.293 

Embedded 

control 

system 

2 

.415 
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What  about  other  possible 
predictors  of  risk  themes? 


Found  no  predictors  of  risk  themes  in  business  goals  or 
domains. 


Have  not  analyzed  based  on  architectural  styles. 


18  is  a  limited  data  set  and  ATAM  does  not  necessarily 
collect  the  correct  information  for  predicting  risk  themes. 


Conjecture:  Organization  setting  is  a  significant  factor  in 
predicting  risk  themes. 
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Recommendations  based  on  what 
is  known  so  far 

Practitioner 

•  Use  checklists  early  in  the  project  to  mitigate  likely  risks 

•  Use  known  techniques  for  mitigating  performance  and 
requirements  volatility  risks. 


Researcher 

•  Explore  hypothesis  that  risks  are  related  to 
organizational  setting 

•  Determine  techniques  to  mitigate  risks  of  organizational 
awareness  and  lack  of  addressing  important 
considerations. 
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ATAM  Evolution 

Initial  thoughts: 

•  Integrate  business  goals  into  utility  tree 

•  Develop  risk  themes  based  on  categories  presented 
here. 


We  welcome  ideas  as  to  how  this  data  can  be  used  to 
improve  the  ATAM  method. 
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More  information 

Categorizing  Business  Goals  for  Software 
Architectures 


Rick  Kazman 
Len  Bass 

Technical  Report 
CMU/SEI-2005-TR-021 

Report  on  risk  themes  in  preparation. 


©  2006  by  Carnegie  Mellon  University 


page  24 


Carnegie  Mellon 

Software  Engineering  Institute 


Questions? 
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